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ABSTRACT 


The Marine Forces Reserve (MARFORRES) comprises mainly “citizen-soldiers” 
who are required to operate across a broad spectrum of military operations at extended 
distances to support the Marine Corps missions. In recent years, the accountability and 
readiness scores for MARFORRES units have been declining. There have been previous 
studies endorsed by MARFORRES G-4 to develop a value model for the Supply and 
Maintenance Team (SMAT) inspection program through the analysis of a stakeholder- 
derived objectives hierarchy. This thesis expands upon previous work by focusing on the 
following: 1) review of the existing value models to ensure they reflect the true value of 
the SMAT inspection program, 2) validation of the value models to articulate the 
contributions that the SMAT inspection program provides to MARFORRES mission, and 
3) demonstration of the utility of value models for resourcing decisions in a budget- 
austere environment. MARFORRES can use the value models to revise its inspection 
policies to focus on value-driven inspection rather than mission-driven inspection. This 


will help to maximize the total value of SMAT inspection to the MARFORRES mission. 
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EXECUTIVE SUMMARY 


Over the past decade, United States Department of Defense (DOD) has recognized 
that the operational tempo of numerous overseas deployments has impacted the operational 
readiness of the United States Armed Forces. This impact was reflected in the decline of 
accountability and readiness scores in the Marine Forces Reserve (MARFORRES) units in 
recent years. The concept of a Supply and Maintenance Assistance Team (SMAT) 
inspection program was introduced in 2013 to evaluate the overall logistical readiness and 
accountability of MARFORRES units. The MARFORRES G-4 (Logistics Directorate) has 
requested the Naval Postgraduate School (NPS) to develop a method to understand the 
contributions that the SMAT inspection program provides to the MARFORRES mission. 
The approach NPS took was to develop value models related to readiness and 
accountability metrics, validate the models, then apply the value models to support value- 


based decision making concerning SMAT inspection scheduling. 


This thesis uses the Systems Decision Process (SDP) framework to examine the 
SMAT inspection program. An initial review of the existing value models from a previous 
study of SMAT operations (Yee 2019) revealed that the existing value models contained 
inconsistencies and ambiguities that could fail confuse interpretations of inspection data, 
thus possibly failing to accurately reflect the value of the SMAT inspection program. This 
thesis modified the existing value models and validated the updated value models with the 
MARFORRES G-4 to more accurately capture the contributions of the SMAT inspection 
program. Lastly, this thesis described how the application of value scores to a scheduling 
optimization model which resulted in more efficient resource usage in a budget-constraint 


environment. 


The development of value metrics began with creating a comprehensive objectives 
hierarchy that accounted for all key objectives that are important to the stakeholders. The 
top-level fundamental objective was identified through stakeholder analysis, and was 
verified with the MARFORRES G-4. The top-level objective is focused on ensuring that 
the MARFORRES units have the right types and numbers of properly maintained 


XV 


equipment. The second level of the hierarchy decomposes the top-level objective into two 
sub-objectives: 1) Improve equipment readiness; and 2) Improve equipment accountability. 
A third original sub-objective “Improve impact on unit processes” was removed from the 
hierarchy based on changes made to the SMAT inspection program concept of operations 
that focus on virtual inspections and no longer requires the units to submit corrective action 


plans. 


Key attributes and associated metrics were developed from the sub-objectives that 
would allow the SMAT to evaluate MARFORRES unit equipment readiness and 
accountability. The quality, utility, or relevance of some metrics made them unsuitable for 
creating useful value models; perpetual inventory status, maintenance process request 
(MPR) status, and the preventive, maintenance, checks and services (PMCS) status were 
removed from the original objectives hierarchy. Figure | illustrates the resulting objectives 


hierarchy of the SMAT inspection program. The overarching objective is in the white box, 


sub-objectives in blue boxes, attributes in red boxes, and metrics in yellow boxes. 


Ensure MARFORRES units have the right types and numbers of properly maintained equipment 





Figure 1. Objective Hierarchy of SMAT inspection program 
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The SMAT program’s revised concept of operations focuses on virtual inspection, 
making supply readiness (SR) and maintenance readiness (MR) from Yee’s (2019) study 
inapplicable as variables. The existing variables of SR and MR ratings are replaced by the 
percentage of PMCS discrepancies as the surrogate variable; approximately 70% of the 
PMCS discrepancies and value scores pairs are compatible. Due to the limited duration and 
the difficult in collecting new data midway through this study, the findings are accepted. 
The PMCS discrepancies provide an indication of the overall maintenance health of the 


organization and guides the development of the value curves. 


This thesis represents the value derived from of each attribute through quantitative 
value modeling. The surrogate variable, percentage of PMCS discrepancies, is plotted 
against each metric to develop the value curves. There are three main shapes that are 
hypothesized in this thesis: 1) Linear return to scale (RTS), 2) Decreasing RTS (concave 
curves), and 3) Increasing RTS (convex curves). The thesis verified the relevance of the 
value curves through theoretical, empirical, and operational justifications. The value curves 
are then used to convert each unit’s metrics captured in the inspection into raw value scores 
(RVS). The RVS are subsequently transformed into weighted value scores (WVS) using 
measure weights that reflect the importance of the metric to the stakeholder. The sum of 
the WVS can then be computed for the total value score of each unit, which represents the 
value that the unit would gain from the inspection and thereby the value that MARFORRES 
would gain from the unit being inspected. The total value score informs the SMAT program 


decisions on where scarce inspection resources should be invested to return the most value. 


This thesis also demonstrated the application of the value models in scheduling 
SMAT inspections of MARFORRES units for a six-month period. A greedy heuristic 
model and Lugo’s (2020) optimization model are used to develop inspection schedules 
with the objective of maximizing the total value, and then the results compared between 
the two models. The greedy heuristic model focuses on selecting locally optimal solutions 
(units with the highest value score) in each month. In general, the greedy heuristic is how 
the SMAT is implementing value scores. The greedy heuristic was able to schedule eight 


units for inspection in the six-month period and attained a total value score of 47.3. In this 


XVii 


schedule, utilization of SMAT resources was approximately 60%. Lugo’s (2020) 
optimization model was able to schedule 16 units for inspection in the six-month period 
and attained a total value score of 84.9. In this schedule, utilization of SMAT resources 
was 100%. Compared to the greedy heuristic model, Lugo’s optimization model is able to 
attain approximately double the total value score. By implementing the value scores in 
conjunction with an optimization program, the G-4 can more efficiently use its resources 
to obtain the set of inspections that present the greatest contribution to the MARFORRES 


mission. 


Although the value models are observed to generally follow a logical pattern, there 
are some exceptions. Due to the limited duration of this study, the exact explanation for 
these anomalies was not discerned. The SMAT program’s revised concept of operations 
means that there may be certain unforeseen confounding factors such as misinterpretation 
and verification of data that are initially difficult to discern. A major recommendation of 
this thesis is to continue collection and analysis of data to improve the consistency and 
robustness of the value models. Lastly, should future developments allow, reinstate the 
sub-objective “Improve impact on unit processes” sub-objective to physically assess the 
degree to which the SMAT inspections improved the units’ processes. This sub-objective 
encompasses another aspect of the SMAT inspection program that focuses on the 
developmental impact. Various metrics such as the percentage of corrective actions 
implemented can be further evaluated to examine the extent to which the efficiency of the 


unit processes has improved. 


XViil 


ACKNOWLEDGMENTS 


First, I would like to express my deepest gratitude to Professor Alejandro 
Hernandez for his guidance, patience and mentorship. His commitment and support have 
enabled me to better understand the systems engineering discipline and complete this 


challenging but rewarding journey at the Naval Postgraduate School. 


Second, I would like to express my appreciation to Mr. Gary Parker for his time 


and advice that has helped to give alternative views to our challenges. 


Third, I would like to thank Mr. Paul Ewing for his assistance in formulating the 


inspection schedule using his optimization model. 


I am also grateful to the Singapore Armed Forces for giving me this opportunity to 


further my studies in Naval Postgraduate School. 


Lastly, I would like to thank my family for being my pillar of strength and support 


throughout this journey. 


Vii 


THIS PAGE INTENTIONALLY LEFT BLANK 


Vill 


I. INTRODUCTION 


A. BACKGROUND 


Logistics is the foundation on which successful military operations are built 
(Wissler 2019). However, past experiences and analysis have shown that the importance 
of logistics is often under-appreciated in most militaries around the world. The 1990s Gulf 
War underscored this attitude about logistics. Although the United States Armed Forces 
demonstrated its ability to transport large amounts of logistics to a theater and sustain its 
logistics readiness, it also faced several initial logistical shortcomings (Conahan 1991). The 
Government Accountability Office (GAO) highlighted that the Marine Corps had issues 
with the management and visibility of prepositioned supplies in the initial phase of the 
operations. The logistical shortcomings meant that the Marine Corps had to depend on their 
organic logistical supplies to maintain an acceptable level of logistical readiness for the 


initial phase of their operations (Conahan 1991). 


The Gulf War also showcased the value of the total force. The Marine Forces 
Reserve (MARFORRES), which was established via the Naval Appropriations Act in 
1916, had its largest mobilization since the Korean War in its effort to support the Marine 
Corps missions (United States Marine Corps Forces Reserve 2016). The MARFORRES, 
which comprised mainly of “citizen-soldiers,” is an integral part of the Total Force. Given 
the requirements of the Marine Corps to operate across a broad spectrum of military 
operations at extended distances with little forewarning, it is crucial for MARFORRES to 


maintain high logistical readiness to sustain its force for mission success. 


In recent years, the accountability and readiness scores for MARFORRES units 
have been declining (Yee 2019). This result has become especially evident in the complex 
operational environments in which the MARFORRES has been most deeply involved. The 
concept of a Supply and Maintenance Assistance Team (SMAT) inspection program was 
introduced in 2013 to gauge the overall readiness and compliance of MARFORRES units. 
The MARFORRES G-4 (Logistics Directorate) requested the Naval Postgraduate School 


(NPS) to develop a set of value models for the SMAT inspection to understand the 
contributions that the SMAT inspection program provides to the MARFORRES mission. 


B. OBJECTIVES 


This thesis examines the inspection system that the MARFORRES G-4 has 
developed to support the MARFORRES mission with three primary objectives in mind. 
First, this thesis aims to review the existing value models to ensure that they reflect the true 
value of the SMAT inspection program. Secondly, this thesis validates the value models to 
articulate the contributions that the SMAT inspection program provides to MARFORRES 
mission. Lastly, this thesis describes the utility of value models for resourcing decisions in 


a budget-austere environment. 


C, RESEARCH QUESTIONS 
In order to meet the thesis objectives, this thesis will answer three critical questions: 


1. How can the SMAT inspection program be revised to evaluate 


MARFORRES’s equipment readiness and accountability? 


2. How can we validate and optimize the value model to articulate the value 


that the SMAT inspection program provides? 


3: How can value models be used to support resourcing decisions? 


D. BENEFITS OF THESIS 


1. The reexamination of the contributions brought by the SMAT inspection 
program will enable MARFORRES stakeholders to adjust their inspection 
policy to focus on value-driven inspection of units that are anticipated to 
have the greatest improvement after inspections. This contrasts with the 
traditional priority-based inspection that focuses on operational and 
logistical commitments, which may be a challenge for SMAT’s limited 
number of available inspectors. The derived value model will allow 
MARFORRES to improve SMAT’s use of its resources to conduct 


inspection. 


The methodology used to develop and validate the value model and its 
associated metrics can be adapted to other logistical inspection programs 
within the United States Armed Forces to reflect the true intent of these 
inspections. This is because while most services have several inspection 
programs that evaluate the state of units’ readiness, there may be an unclear 
delineation among the various inspection programs. Having well-developed 
value models will aid in effectively using inspection and assistance 
resources to provide a comprehensive analysis of the units’ logistical status 


and readiness to support their mission. 


E. ASSUMPTIONS AND LIMITATIONS 


The scope of this thesis is bounded by the following assumptions and limitations: 


hs 


The metrics formulated can be used to quantitively assess the performance 


in achieving the objective. 


The value functions for each metrics reflect the stakeholders’ perception of 


the contribution that the inspection provides to the MARFORRES mission. 


The value is defined as the anticipated improvement of the unit post- 


inspection. 


The mode of data collection is homogenous across the units inspected 


virtually. 


All units will strive to prepare well for the inspection. 


F. ORGANIZATION OF STUDY 


Chapter II focuses on a literature review that informs this work and reflects the 


current concept of the value model design based on stakeholder’s analysis and objectives 


hierarchy. The key findings of the existing value model will be highlighted. Lastly, the 


value of inspection programs in industry and its metrics will be explored. Chapter III 


focuses on the validation process and the method used to revise the existing value model 


design. Chapter IV analyzes the data and validates the revised value model. Chapter V 


concludes the thesis by articulating the revision made to the existing value model design 
and highlights the value that the SMAT inspection program provides to MARFORRES. 
Recommendations for incorporating the value models for resource decisions are a natural 
extension of this work and are provided. Additionally, areas for further study are identified 


that will continue to enhance the value models and their usefulness to MARFORRES. 


I. LITERATURE REVIEW 


This chapter examines archived information that helps to form the methodology 
and analysis of this thesis. It begins with an overview of the organization of the 
MARFORRES and the SMAT inspection program. This chapter then examines the concept 
of readiness and accountability, followed by a discussion of the characteristics of good 
measures of effectiveness (MOEs). This chapter explains the formation of the objectives 
hierarchy and value model. Lastly, this chapter describes the study approach which 


includes value-driven decision making and optimization. 


A. ORGANIZATION OF MARFORRES 


The organization of MARFORRES provides a basic understanding of its missions 
and structure which helps to identify stakeholders with vested interests in the SMAT 
inspection program. The MARFORRES mission is to “...augment and reinforce active 
Marine forces in time of war, national emergency or contingency operations, provide 
personnel and operational tempo relief for the active forces in peacetime, and provide 


service to the community” (United States Marine Corps Forces Reserve 2020). 


MARFORRES’s organizational structure is similar to the Marine Expeditionary 
Force (Feickert 2014). The Commander MARFORRES (COMMARFORRES) exercises 
command and control of the Major Subordinate Commands (MSCs) through the 
General/Special Staff to fulfill their mission profiles. The General/Special Staff 
organization reflects the functions as the staff sections perform; G-1 Administrative and 
Personnel Section, G-2 Intelligence, G-3/5 Operations and Plans, G-4 Installations and 
Logistics Section, G-6 Communications, Command Inspector General, and the G-8 
Requirements Section. The section of interest to this thesis is the G-4 Installations and 


Logistics Section, where the SMAT program resources reside that inspect the MSCs. 


B. ORGANIZATION OF SMAT 


The organization of SMAT provides a basic understanding of its mission and 


concept of operations. The concept of operations offers an insight into how the inspections 
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are conducted and the types of data collected. The SMAT mission is to “...conduct analysis 
on MSCs to ensure Marine Corps orders, policies, and directives are adhered to in order to 
support accountability, readiness, and readiness reporting to the force” (United States 


Marine Corps 2016). 


SMAT is made up of subject matter experts with knowledge and expertise in 
logistics readiness and accountability. The SMAT uses two modes of analysis, namely: 1) 
formal inspections, and 2) site-assist visits. When a unit is selected for formal inspection, 
the SMAT first conducts a virtual inspection using the Maintenance Management Tool 
(MMT). The MMT is used to retrieve inputs from Marine Corps Global Combat Support 
System (GCSS) records to determine the unit’s current condition. After completing the 
virtual inspection, the formal on-site inspection will then be conducted on the unit’s Supply 
and Maintenance sections. During the on-site inspection, the unit’s supply and maintenance 
processes, procedures, accountability, and record keeping are examined for compliance 


with applicable regulations and directives. 


Prior to the on-site inspection, a pre-determined sample of records and items are 
selected for inspection. After the on-site inspection, a formal report is produced within 
approximately 30 days for any deficiencies found, and corrective recommendations are 
made to the unit. The unit is then required to provide a corrective action plan to G-4 based 
on the equipment discrepancies identified in the inspection performance report (United 
States Marine Corps 2016). Depending on the severity of the deficiency, SMAT will re- 
inspect the unit to ensure corrections have been made. For the site-assist visits, SMAT 
conducts an informal inspection meant to train and guide requesting units that need 
assistance in ensuring compliance with MARFORRES orders. However, formal reports for 


site-assist visits are not produced unless requested by the MSC. 


The SMAT inspection program has reviewed its concept of operations as of Fiscal 
Year (FY) 2020, focusing solely on virtual inspections using MMT for its analyzes. This 
is also influenced by the outbreak of 2019 Novel Coronavirus (COVID-19), in which the 
Marine Corps had to review its mission essential functions to restrict the movement of its 


personnel in accordance with the Marine Corps Disease Containment Preparedness 
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Planning Guidance (United States Marine Corps 2020). This unforeseen circumstance 
further promotes the implementation of virtual inspections. Additionally, the new 


guidelines no longer require the units to submit corrective action plans. 


C. READINESS AND ACCOUNTABILITY 


Readiness and accountability form the cornerstones of the SMAT’s mission 
statement and articulate the purpose of the SMAT’s operations. The mission statement 
provides the direction on where the organization is striving towards and assists in making 
informed decisions that will be beneficial to the organization. It is therefore crucial to 
understand the specifics of these two aspects of readiness and accountability in order to 


gain clarity on the focus of the SMAT inspection program. 


The United States Department of Defense (DOD) defines operational readiness as 
“the capability of a unit/formation, ship, weapon system, or equipment to perform the 
missions or functions for which it is organized or designed” (Office of the Chairman of the 
Joint Chiefs of Staff 2020). Operational readiness is decomposed into three parts: 1) 
manning, 2) equipment, and 3) training. Equipment readiness can be defined as the 
influence of maintenance and parts availability on the equipment’s operating status 
(Rumbaugh 2017). This aspect is crucial as the MSCs need to have the right type of 


properly maintained equipment to be operational ready. 


Accountability is defined by DOD (2020) as “the obligation imposed by law or 
lawful order or regulation on an officer or other person for keeping an accurate record of 
property, documents, or funds.” In the realm of logistics, it can be elaborated as the tracking 
and maintaining of proper records for the equipment assigned to the MSCs. This 
emphasizes on the importance of documentation, which allows MARFORRES to have an 


oversight on its equipment status to support decision-making. 


Understanding the two objectives of the mission statement will therefore allow us 
to understand the focus of the SMAT inspection program. This will facilitate the conduct 


of the preliminary requirement assessment of the SMAT inspection program in conjunction 


with the stakeholder analysis to develop the value models that reflect the true value of the 


SMAT inspection program to support the MARFORRES mission. 


D. MEASURE OF EFFECTIVENESS 


MOEs are an essential part of systems engineering that help establish how well a 
system achieves a set of specific mission requirements. Despite the importance of MOEs, 
there is no one universally accepted definition for the term. The International Council on 
Systems Engineering (INCOSE) (2015) added that MOEs should be evaluated “in the 
intended operational environment under a specified set of conditions.” DOD (2020) 
emphasized that the metrics must be used to measure a current system state and “with 


change indicated by comparing multiple observations over time.” 


In order to establish the contribution of the SMAT in its support to the 
MARFORRES mission, the proposed model must have well-defined metrics that 
realistically represent the status of the units’ equipment readiness and accountability. 
Sproles (2000) suggested that the development of these metrics should be focused on the 
stakeholder needs and requirements with the standards established by the stakeholders. 
INCOSE (2010) describes characteristics of a good metric as such: 1) Relevance, 2) 
Completeness, 3) Timeliness, 4) Simplicity, 5) Cost-Effectiveness, 6) Repeatability, and 
7) Accuracy. The following three characteristics are adapted for the development of the 


metrics in this thesis. 


1. Relevance 


The developed set of metrics must be relevant to the mission with clear descriptions 
of their functions (Miller et al. 2010). Sproles (2000) added that the development of the 
metrics should be focused on the stakeholder needs and requirements. Sproles (2002) 
further explained that metrics must tackle the operational issues, which are the “emergent 
properties that the system must have in order to perform its function.” This will ensure that 
the identified metrics are pertinent to the objectives of the thesis and agreed upon by the 


stakeholders. 


This thesis identifies the key value proposition through stakeholder analysis and 
develops the fundamental objective to promulgate the metrics for the value models. The 
SMAT inspection program can then use these metrics to collect relevant information during 
the inspections. The information collected will allow MARFORRES to understand how 
the SMAT inspection program contributes to make informed decisions on its use of the 


SMAT resources. 


2; Completeness 


The developed set of metrics must be comprehensive and account for all the key 
objectives of importance to the stakeholders (Miller et al. 2010). The relative importance 
of each metrics is be based on the stakeholder analysis and applied through swing weights. 
In this study, the completeness of the metric specifications ensures their reliability and the 
total quality of the SMAT value models. This will provide assurance that the model covers 


all angles and perspective of the stakeholders. 


Kirkwood (1997) established that the metrics on the same tier within the hierarchy 
should not overlap in order to prevent redundancy. In a study conducted to model the 
selection of automobiles, research has shown that it is crucial that the metrics used in the 
model are not confounded by unintended correlated measures (Cook and Wissmann 2007). 
These correlated measures may unintentionally have an impact on the true value of the 
SMAT value models. Therefore, it is vital to identify a comprehensive set of metrics for 


the value models which conform to the relevant objectives. 


3. Accuracy 


The developed set of metrics must be specific and measurable. The accuracy of the 
metrics can be defined as the extent to which their numeric values truly represent the entity 
being measured (Miller et al. 2010). Keeney (1992) added that a good metrics should be 
able to provide a defendable method in collecting and assessing its performance. 
Furthermore, a measurable metric will help prevent ambiguity of interpretation or loss of 


information which will have an impact on the accuracy of the information collected. 


Given that the SMAT inspection program has transitioned to virtual inspection in 
FY 2020, the SMAT inspectors no longer conduct formal on-site inspections. The readiness 
levels of the units are assessed virtually based on the individual units’ GCSS records 
accessed through the MMT. Therefore, the specific and measurable metrics are even more 
crucial for the collection and dissemination of information between the various entities 
involve in the evaluation process. Having specific and measurable metrics helps prevent 


any ambiguity of interpretation and loss of information between the various entities. 


E. OBJECTIVES HIERARCHY 


Keeney (1992) explained that value-driven thinking placed emphasis on the proper 
structuring of decisions via a fundamental objective. It is therefore essential to identify the 
fundamental objective important to the stakeholders for decision making. This fundamental 
objective can then be translated into value objectives that can meaningfully established an 
objectives hierarchy. This forms the basis of the system’s requirement that can be broken 


down into sub-objectives and their respective metrics for evaluation of the system. 


As an example, an objectives hierarchy for the development of a rocket system is 
shown in Figure 1. Based on the illustration, the fundamental objective is “to develop a 
rocket that delivers payloads rapidly and accurately to the intended destination” (Parnell, 
Driscoll and Henderson 2010). The fundamental objective is decomposed into its functions 
and relevant objectives based on the values articulated by the stakeholders, which in this 
case is to “maximize mobility of the launch platform” and “minimize logistics footprint” 
(Parnell, Driscoll and Henderson 2010). The formulated metrics provide a quantitative 
means to analyze the extent to which the rocket system achieves the objectives, such as the 
speed of the launch platform and the percentage of platform that can traverse to deduce the 


mobility of the launch platform as part of its objectives. 
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Fundamental Objective Deliver payload 
rapidly and accurately 


Functions 
Launch rocket 


Objectives Maximize mobility of Minimize logistics 
launch platform footprint 


Speed of launch Number of operators 
Value Measures platform (km/hr) (#) 


% of platform that can 
traverse (%) 





Figure 1. | Objectives hierarchy of a rocket system. Adapted from 
Parnell, Driscoll, and Henderson (2010). 


F. VALUE MODEL 


Quantitative value modeling enables the assessment of the effectiveness of the 
various solutions based on the stakeholders’ values. This is done by constructing the value 
curves and applying the swing weights for each of the identified objectives. The value 
curves can be constructed through the relationship between the metrics with a discrete or 
continuous scale as reflected in the y-axis. Moreover, Parnell (2010) emphasized that the 


y-axis must be strictly consistent for all the metrics. 


The shapes of the value curve established by the metrics can then be classified 
under four general forms: 1) Linear return to scale (RTS), 2) Decreasing RTS (concave 
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curves), 3) Increasing RTS (convex curves), and 4) S-curve as shown in Figure 2 (Parnell, 


Driscoll and Henderson 2010). 


Linear returns to scale 


(RTS) 


Decreasing RTS 
(concave) 


Increasing RTS 
(convex) 


S Curve 





Figure 2. Shape of value curves. Source: Parnell, Driscoll, 
and Henderson (2010). 


1. Linear RTS 


A linear RTS indicates a proportional return of value between the variables within 
the range of interest. In order to assess if the variables of the value curve have a linear 


relationships between them, Pearson’s correlation coefficient is used to measure the 


12 


strength of their linear relationship (Hinkle, Wiersma and Jurs 2003). The absolute value 
of the correlation coefficient has a range 0 to 1, where a value of 0 denotes no linear 
correlation while a value of 1 denotes a perfect correlation. The interpretation of the 


strength of the Pearson’s correlation coefficient is shown in Table 1. 


In this thesis, an absolute correlation coefficient value of 0.5 and above, which 
minimally denotes a moderate relationship, would be sufficient to conclude that there is a 
linear correlation between the identified variables. Given that the value has been defined 
as the anticipated improvement of the unit post-inspection, the shape of the value model 
should exhibit a positive correlation with respective to the number of discrepancies that the 


SMAT can uncover during the inspection. 


Table 1. The strength of Pearson’s correlation coefficient. Source: Hinkle, 
Wiersma, and Jurs (2003). 
































Absolute Value of r Strength of Relationship 
r<0.3 Negligible 
0.3<r<0.5 Low 
0.5<r<0.7 Moderate 
0.7<r<09 High 
(209 Very High 
2. Decreasing RTS (Concave Curve) 


The decreasing RTS is associated with the law of diminishing returns, in which the 
output increase at a decreasing rate (Buede 2009). This means that the proportionate 
increase in the inputs does not lead to an equivalent increase in output. As an example, a 
study from the Department of Mechanical Engineering, Polytechnic University of Milan, 
examined the relationship between the probability of failure and the length of inspection 
interval of various types of cracks found on railways components (Carboni and Cantini 


2016). The proposed ultrasonic inspection procedure aimed to quantify the result as part of 


13 


its dedicated inspection plan to optimize the inspection intervals and ensure the safety 


service of safety critical mechanical components. 


The relationship is shown in Figure 3, where the graph adopts a concave curve in 
which probability of failure increases gradually to a certain threshold value as the length 
of the inspection interval gets wider. This quantitative assessment provides insights into 
the formulation of the shape of the SMAT value model. It is essential for SMAT to be 
aware that for a decreasing RTS, there is a maximum to which the stakeholder can extract 
from the inspection, beyond which any additional resources such as time and manpower 


will yield little or no additional value. 


——— Semuorcuser crack - 1.57 awn 
—— Shallow crock - 7 65 mm 
~ow~rn Gallow crack - 157 mm 


7.50x10° 1.00x10° 1.25x10° 1.50x10° 1.75x10° 


Length of the inspection interval [km] 





Figure 3. Probability of failure and inspection interval of railways 
components. Source: Carboni and Cantini (2016). 
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3. Increasing RTS (Convex Curve) 


The increasing RTS is associated with economies of scale, which focuses on 
production where the output increases by a larger proportion as the input increases and 
reflects the marginal productivity with which resources are deployed (Suranovic 2010). An 
example of such a relationship is shown in a study conducted by the Department of 
Mechanical Engineering, Polytechnic University of Milan, which analyzed the fatigue 
crack growth lifetime estimations (Carboni and Cantini 2016). The relationship between 
crack dimensions and the normalized life of the railways components is shown in Figure 
4. The graph adopts a convex curve relationship where the propagation of crack increases 


by a larger proportion as its normalized lifetime increases. 
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Figure 4. Fatigue crack growth lifetime estimations of railways components. 
Source: Carboni and Cantini (2016). 
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This type of quantitative assessment also provides insights into the formulation of 
the shape of the SMAT value models. Increasing RTS in economics is an indicator that a 
firm can produce its output efficiently and thereby has the capability to undertake future 
growth (Suranovic 2010). In the context of SMAT inspection, this would mean that 
increase in the number of discrepancies that the SMAT uncovers during an inspection 
would provide a greater proportion of value in terms of the anticipated improvement of the 


unit post-inspection. 


4. S-Curve 


The S-curve reflects a combination of increasing and decreasing RTS, where the 
maximum and minimum performance preference is within the range of interest (Buede 
2009). The S-curve indicates an initial exponential growth rate and subsequently adopts a 
linear profile before reaching saturation as per the law of diminishing returns. The S-curve 
can be formulated using the logistic function which considers the maximum capacity of 


growth the model can achieve. 


In the context of SMAT inspections, the logistic function means that limited value 
can be extracted unless significant changes and resources are committed. The SMAT has 
optimize the usage of its resources to conduct inspections to assess the units’ logistics 
readiness. Due to the limited number of available inspectors, there is a maximum to the 
value which the stakeholder can extract from the inspection, beyond which any additional 


resources such as time and personnel staffing will yield little or no additional value. 


G. OPTIMIZATION OF SMAT INSPECTION 


Given the limited resources coupled with competing interests from various 
inspection requirements, the SMAT inspection program is aware that it needs to improve 
its usage of resources to conduct inspections that will be the most beneficial to the 
MARFORRES mission. There are many different methods to optimize a schedule, such as 
a greedy heuristic model or using a mathematical integer model. The greedy heuristic 
model is a simple and intuitive method that works on the principle of selecting locally 


optimal solutions at each decision point of the optimization program (Brilliant 2020). The 
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mathematical integer model focuses on objective functions and constraints to optimize a 
schedule. Furthermore, this model is highly useful for practical applications that involves 


multiple variables and constraints. 


As part of an optimization study for inspection scheduling, Lugo (2020) developed 
a multi-period mixed-integer linear program using the General Algebraic Modeling System 
(GAMS) to maximize the total number of higher priority units within a 36-month period, 
while satisfying various resource and budget restrictions. Based on Lugo’s model, the units 
are quantified using various variables such as the inspection prioritization criteria. The 
prioritization criteria depend on the MARFORRES’s operational and _ logistical 
commitments, where units scheduled for deployment or assigned mobilization to support 
active duty force are assigned a higher score. The priority score ranges from one to five, 
with priority one denoting the highest priority units and priority five denoting the lowest 
priority units. Both the greedy heuristic and Lugo’s optimization model provide the 
foundation to use the results derived from the value models to develop a schedule that 


maximizes the total value of SMAT inspection program. 


H. APPROACH TO VALUE MODELING 


DOD recognizes that numerous overseas deployments over the past decade has 
reduced the operational readiness of the United States Armed Forces. The GAO has 
attributed the low readiness level to “emerging and continued demands on their forces, 
reduced force structure, and increased frequency and length of deployments” (Pendleton 
2016). The Marine Corps specifically identified the shortfalls in training resources, such as 
the availability of aircraft as one of the challenges they faced in rebuilding readiness. DOD 
has therefore prioritized improving the readiness level of its forces as part of its mission to 


deter war and to protect the security of the country. 


DOD has undertaken efforts to transform its Global Force Management from 
combatant-demand to a resource-informed process to address the decline in readiness. As 
part of its analysis of the readiness rebuilding effort, the GAO (2016) recommended the 


services to establish goals and associated metrics to evaluate and validate their readiness 
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efforts to meet the mission requirements. This shows the emphasis placed on value-driven 


decision-making. 


Yee (2019) formulated an objectives hierarchy based on MARFORRES 
stakeholders as part of his research into the development of a value proposition for the 
SMAT inspection program. He formed a hypothesis that the value of an inspection is 
related to the inspected unit’s response to the SMAT findings and will therefore improve 
supply readiness (SR) and maintenance readiness (MR) ratings after 90 to 120 days of the 
inspections. The shapes of the value curve for each metrics are subsequently constructed 
using the SR and MR ratings as a guide for the shape of the value model. MR ratings are 
used to evaluate metrics that fall under the sub-objective to “improve equipment 
readiness,” and the SR ratings are used to evaluate the metrics that fall under the sub- 


objective to “improve equipment accountability.” 


Yee (2019) concluded that some of the empirical analysis proves to be difficult, 
and as such, theoretical and operational justifications were used to approximate the shapes 
of metrics with weak correlations. He recommended that the existing value model should 
be examined and revised to provide a better evaluation of MARFORRES’s supply and 
maintenance readiness. A potential method may be to identify a surrogate variable that can 
be used to capture the value of the metrics. However, a critical issue is identifying the best 
way to assess the feasibility of the surrogate variables against the currently available 


metrics to improve the quality assessment of the value model. 
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Hl. METHODOLOGY 


This thesis adopts the Systems Decision Process (SDP) framework of Parnell, 
Driscoll, and Henderson (2010) which provides a collaborative, dynamic, and value-driven 
decision process. Throughout the SDP framework, there is a strong emphasis on continuous 
engagement with the stakeholders to identify their concerns, in order to formulate a 
coherent value model. The SDP framework consists of stakeholder analysis, objectives 
hierarchy and formulation of the value model as shown by an IDEF2 model in Figure 5. 
Verification and validation are be conducted to ensure that the value models fulfill their 
intended purpose. Lastly, the impetus for optimization of the SMAT inspection schedule 
is discussed to maximize the total value of SMAT inspection to the MARFORRES mission. 


System boundary 


requirements 





Stakeholder list Stakeholder's 
Initial problem Analysis 
statement At 


Stakeholders Objectives 
Hierarchy 


Ai2 


Stakeholder needs 





Refined problem 
Value Model statement 
Screening criteria 
A13 Quantitative value 
model 





Figure 5. IDEF2 representation of the problem definition phase of the SDP. 
Adapted from Parnell, Driscoll, and Henderson (2010). 
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A. STAKEHOLDER ANALYSIS 


The stakeholder analysis forms the basis of deriving the fundamental objective, 
functions, and constraints of the system. It is crucial to identify key interests and concerns 
of the stakeholders since they may have different perspectives on the overall system (Buede 
2009). Given that managing stakeholder expectations is intrinsic to the success of the 
system, attributes have been established to prioritize stakeholder’s concerns based on the 
stakeholder’s power to influence the system, and the stakeholder’s interest in the system 


(Mayers 2005). 


The result of the previous stakeholder analysis conducted is revisited through 
teleconference and email correspondence to understand any changes in their perspective, 
functions or concept of operation. This reexamination identifies the key value proposition 
which captures the concerns of the different groups of stakeholders based on the established 


stakeholders’ prioritization attributes. 


B. OBJECTIVES HIERARCHY 


The objectives hierarchy is a qualitative model of the stakeholder’s valuation of the 
system’s behavior or performance. This hierarchical model is critical for assessing and 
evaluating the value of the inspection program to the MARFORRES mission. Chapter II 
notes that the objectives hierarchy of a system contains a pictorial representation of the 
objectives and the relevant metrics that are important to the system’s stakeholders for 
decision-making (Buede 2009). Through the stakeholder analysis, the top-level 
fundamental objective can be defined with the identification of the stakeholders’ key value 


proposition. 


The fundamental objective can be subsequently decomposed into its sub-objectives 
and their metrics as shown by illustration of the rocket development program in Chapter 
Il. This thesis uses the characteristics of relevance, completeness, and accuracy to support 
the metrics that reflect the objectives hierarchy. There must be well-defined metrics that 
realistically represent the status of the unit’s equipment readiness and accountability. This 


gives the assurance that the model reflects the true interest of the stakeholders towards the 
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value models. The objectives hierarchy also serves as a qualitative value model that will 
be used to build the quantitative value model to evaluate to what extend the metrics meet 


the fundamental objective. 


C. VALUE MODEL 


Quantitative value modeling focuses on constructing the value curves and applying 
the swing weights to each of the identified objectives. Chapter II discusses how the value 
curves can be constructed through the relationship between the metrics. Yee’s (2019) 
previous study has highlighted that using supply and maintenance readiness ratings may 
not be an ideal choice due to the inconsistency in evaluating the value model and the 
ambiguity in interpreting the function of supply and maintenance readiness. This thesis 
extends Yee’s (2019) work by identifying a surrogate variable that can be used to guide 


the development of the value curves. 


Multivariate analysis was conducted on the nine existing metrics based on the 
inspection findings of 23 units. Based on the analysis, the author has identified the 
percentage of Preventive Maintenance, Checks and Services (PMCS) discrepancies as the 
best candidate for the surrogate variable. The variable can be computed based on the 
percentage of overdue PMCS against total equipment that required PMCS in the 
MARFORRES PMCS report. The last step in value modeling is to apply swing to develop 
the value functions. The swing weights depend on the relative importance of each metrics 
to the stakeholders and their variation within the range of the metrics (Parnell, Driscoll, 
and Henderson (2010). The value modeling process provides the framework to evaluate 


the total value of the SMAT inspection program to the MARFORRES mission. 


D. VERIFICATION AND VALIDATION 


INCOSE (2017) defines verification as a set of activities that “compares a system 
or system element against the required characteristics.” It is further added that verification 
may consist of “specified requirements, design description, and the system itself.” 
Verification addresses whether the system was built right. INCOSE (2017) defines 


validation as the set of activities that ensures the system can “accomplish its intended use, 
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goals, and objectives in the intended operational environment.” Validation addresses 
whether the right system has been built in accordance with the stakeholders. For this thesis, 
the system refers to the value models formulated to address the contribution of SMAT 


inspection while the requirements refer to the objectives developed. 


Verification and validation are inherently incremental processes that occur 
throughout the various phases of SDP. Michael’s (2017) continuous validation and 
verification process is applied in this thesis as illustrated in Figure 6. Due to the incremental 
nature of the validation and verification process, the focus is initially on the objectives, 
followed by the design of the sub-objectives and MOEs, and subsequently the formulation 


of the value curves. 


Stakeholders’ needs st Sub-objectives and ' 





Figure 6. A continuous validation and verification process. Adapted from 
Ryan and Louis (2017). 


First, the fundamental objective and its sub-objectives must be verified through the 
stakeholder analysis. Next, relevant MOEs developed must also be verified to be relevant, 
complete and accurate. This will ensure the completeness of the objectives hierarchy and 
ensure that the objectives of the thesis study are met. Third, the value curves formulated 
must be verified through theoretical and operational justifications. The data collected from 
the various units must conform to the specifications as defined by the MOEs. The shapes 


of the value curves between the MOEs must be theoretically justified as reflected in 
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Chapter II. Furthermore, the relationship between the surrogate metric and MOEs must be 


justified based on understanding of the operational procedure of the MARFORRES units. 


Lastly, the post-inspection data of each inspection units’ percentage of PMCS 
discrepancies is collected 30-90 days after their virtual inspection. These data should show 
improvement in the units’ performance to verify that value models developed meet the 
objectives. In terms of validation, the system validation process adopts a iterative nature 
and provide continuous feedback on how well the requirement meets the stakeholders 
needs and expectations, and how well the value models serve their intended purpose in 
articulating the contributions that the SMAT inspection program provides to 


MARFORRES mission. 


E. OPTIMIZATION OF SMAT INSPECTION 


Chapter II notes the current emphasis on MARFORRES’s traditional operational 
and logistical commitments as a basis to develop the SMAT inspection schedule. It also 
discusses the different methods such as greedy heuristic model and Lugo’s (2020) 
optimization model which are simple and intuitive. This thesis seeks to demonstrate how 
value modeling can be applied to both the greedy heuristic and Lugo’s optimization model 
to develop the SMAT inspection schedules for a six-month period. The aim is to maximize 
the total value that can be derived from the inspections in order to improve the SMAT 


program use of its resources. 


Given that the SMAT requires two weeks to inspect each unit, the manpower 
resources required to inspect a unit and the SMAT manpower resources available to 
conduct the inspections are defined as the Full Time Equivalent (FTE) requirements and 
availability from Lugo’s (2020) model. Both the greedy heuristic and the Lugo’s 
optimization model would be computed based on the value score generated from the value 
model in this thesis. The greedy heuristic model prioritizes each unit based on the value 
score generated and chooses the unit with the highest value with regard to the SMAT 
resources available for that month. The greedy algorithm is a myopic view of the schedule, 


only considering the current month. The heuristic would decide from month to month, 
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regardless of what resources are unused in the previous month and what resources may be 


required in the next month. 


Lugo’s optimization model also prioritizes each unit based on the value score 
generated. However, it also focuses on maximizing the utility of the inspectors available 
by including units that may require less resources to inspect. Both model focus on value- 
drive inspection than mission-driven inspection and serve as a basis to maximize the total 


value of SMAT inspection to the MARFORRES mission. 
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IV. ANALYSIS OF RESULTS 


This chapter examines the data collected and the resultant value models, and provides 
analyses of the value score for any given unit. The analysis follows the SDP framework 
introduced in Chapter III. This chapter begins with the review of the stakeholder’s needs and 
concerns. Next, the objectives hierarchy is evaluated to provide a qualitative assessment of the 
value models. Lastly, the formation of the value functions and their application on inspection 


schedule are discussed to articulate the contribution to MARFORRES mission. 


A. STAKEHOLDER ANALYSIS 


In Chapter II, the importance of stakeholder’s power and interest is emphasized to 
assist in the formulation of the value models. The stakeholder’s power to influence is 
defined as the ability to “persuade others into making decisions” on the inspection program 
based on the hierarchy structure of the stakeholder’s organization (Mayers 2005). The 
stakeholder’s interest is defined as the “aspirations or concerns” of the outlook of the 


inspection program (Bryson, Patton, and Bowman 2011). 


The stakeholder’s power and influence are essential to balance their needs 
appropriately given their different competing interests and the limited number of 
inspectors. After revisiting the previous stakeholder analysis, the prospective stakeholders 
involved in the SMAT inspection program are identified as such: 1) COMMARFORRES, 
2) MARFORRES G-4, 3) Inspectors, and 4) MSCs. The key value propositions which 


captured the influence and concerns of the different stakeholders are shown in Table 2. 
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Table 2. 


Stakeholder matrix. Adapted from Yee (2019). 








Stakeholder Power of Influence Interest 
e High. Able to provide overall Understand the resources 
leadership and influence on need to achieve the 
all aspects of policy. MARFORRES mission. 
COMMARFORRES 


Optimize the use of resources 
to achieve MARFORRES 
mission. 





MARFORRES G-4 


High. Able to provide 
leadership and influence on 
guidelines. 


Justify the resources needed 
for the SMAT inspection 
program to support the 
MARFORRES mission. 


Differentiate the SMAT 
inspection program from the 
FSMAO program. 





Medium. Able to influence 


Find the areas for 








on nature of — support, improvement and areas that 
Inspectors ; : : 
implementation, and are not performing well. 
coordination. 
e Medium. Responsible for the Identify the value that the 
MSCs outcome of the inspection. SMAT inspection program 








contributes to the 


performance of the unit. 





A power interest matrix is also utilized as part of stakeholder mapping to provide a 


visual representation of the stakeholder analysis. As show in Figure 7, the two-by-two 


matrix has a vertical axis that represents the stakeholder’s power to influence and a 


horizontal axis that represents the stakeholder’s interest (Bryson, Patton, and Bowman 


2011). Each of the four quadrants of the matrix provides an indication of the level of 


stakeholder prioritization required which will also influence the mode and style of 


communication. 
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Keep satisfied Key players 


Marine Forces Reserve Command ¥*€ 


MARFORRESG-4 * 


Inspectors y& 


Minimal effort Keep informed 


Major Subordinate Commands *& 











Level of interest 








Figure 7. SMAT power interest matrix. Source: Yee (2019). 


Based on Figure 7, the COMMARFORRES and MARFORRES G-4 are located at 
the top right-hand side of the power interest matrix, given that they have the highest power 
of influence on the policy and interest in the success of the inspection program. By contrast, 
the MSCs is located at the bottom right-hand side of the power interest matrix, which 
indicates that they do not have significant influence on the policy but continue to have a 
high interest in inspection program. This is rightfully so given that the MSCs have limited 
influence on the decision and are responsible for the outcome of the inspection. With the 
completion of the stakeholder analysis, the objective hierarchy can then be developed based 


on the matrix and materials discussed in Chapter II. 


B. OBJECTIVES HIERARCHY 


There is a need as part of the SDP framework for continuous engagement with the 
stakeholders to verify the relevance of the existing top-level fundamental objective, sub- 


objectives, and their respective metrics. This iterative approach is conducted through 


at 


multiple teleconference and email exchanges with the stakeholders to solicit inputs to 
refine the existing objectives hierarchy. Through the stakeholder analysis, the top-level 
fundamental objective remains focused on ensuring that the MARFORRES units have the 


right types and numbers of properly maintained equipment. 


The second level of the hierarchy continues to focus on MARFORRES G-4’s 
interest. The sub-objectives are revised from the existing objectives hierarchy to the 
following: 1) Improve equipment readiness; and 2) Improve equipment accountability. The 
sub-objective “Improve impact on unit processes” was removed since the SMAT has 
changed its concept of operations to focus on virtual inspections and no longer requires the 
units to submit corrective action plans. The outbreak of COVID-19 also limits the SMAT 
program’s ability to physically assess the degree to which the corrective action plans 
improve the units’ processes. The sub-objectives are subsequently used to identify the key 
attributes and associated metrics that would allow the SMAT to evaluate the equipment 


readiness and accountability of the MARFORRES units. 


The quality, utility, or relevance of some variables made them unsuitable for 
creating the value models. As a result, perpetual inventory status, maintenance process 
request (MPR) status, and PMCS status were omitted from the existing objectives 
hierarchy. The perpetual inventory status is the percentage of total errors in recording of 
the table of authorized material control numbers (TAMCN) serialized equipment against 
the total number of equipment in perpetual inventory. Based on the data analysis of the 
percentage of perpetual inventory discrepancies, the team observed limited variations 
across the 23 units that are inspected. Consequently, there would be little utility in using 
the perpetual inventory status to delineate that value of inspecting one unit in lieu of 


another. 


The MPR status is the percentage of equipment incorrectly processed for 
maintenance against the total number of maintenance requests. Discussion with the SMAT 
verified that the attribute was not relevant to the stakeholder’s needs. Lastly, the PMCS 
status is removed from the existing objective hierarchy since it will serve as the surrogate 


variable for indicating the overall maintenance health of the organization and guides the 
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development of the value curves. The importance of this surrogate variable is elaborated 
in the next section. Figure 8 illustrates the resulting objectives hierarchy of the SMAT 
inspection program. The overarching objective is in the white box, sub-objectives in blue 


boxes, attributes in red boxes, and metrics in yellow boxes. 


Ensure MARFORRES units have the right types and numbers of properly maintained equipment 





Figure 8. | Objective Hierarchy of SMAT inspection program 


C. VALUE MODEL 


In Chapter II, the team identified the percentage of PMCS discrepancies as a 
surrogate variable to develop the shapes of the value curve. Theoretically, the strength of 
the relationship between the metrics and the surrogate variable would be measured with 
the correlation coefficient. When there are strong correlations, the metrics would follow a 
linear relationship with the surrogate variable. However, when there are negligible or low 
correlation, supporting theories and operational justifications are used to articulate the 
relationship between the metrics and the surrogate variable. The relationship then informs 


how the value scores should behave as the metric changes. 
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For linear RTS, the linear function “y = a + bx” has been adopted, where y and x 
are the dependent and independent variables respectively, a represents the constant term 
and b represents the rate of change of the dependent variable coefficient as the independent 
variable changes (Columbia 2020). For decreasing RTS, the concave curve function “y = 
log(x-h) + k’ has been adopted, where y and x are the dependent and independent variables 
respectively, H and k represent the horizontal and vertical shift factors respectively (Lumen 
2020). Lastly for increasing RTS, the convex curve function “y = a b*’ has been adopted, 
where y and x are the dependent and independent variables respectively, a represents the y 


intercept and b represents the growth factor (Lumen 2020). 


1. Value Curve for Equipment Modification Status 


The metric of equipment modification status focuses on capturing the percentage 
of equipment modification discrepancies found during the SMAT inspections. 
Operationally, the purpose of modification instruction enforces timely equipment 
modification requirements to correct safety and operational deficiencies of the equipment. 
This metric reflects the operational status of the units’ equipment which ultimately 


influences their equipment readiness. 


The value curve assumes a concave shape based on the relationship between the 
metric and the surrogate variable, where the percentage of PMCS discrepancies increases 
at a decreasing rate with the metric scores. This is similar to the concept of decreasing RTS 
as discussed in Chapter II. Based on the empirical data, the metric shows little correlation 
relationship with the surrogate variable. Moreover, the data points of the units inspected 
are concentrated at below 11% of the metric score. This would mean that the added value 
from the findings beyond 11% diminishes rapidly. Based on the law of diminishing return, 
there is a maximum range of values from which stakeholder can extract from the 


inspection. 


As more modification discrepancies are discovered during the inspection, up to a 
point there would be more value derived from the inspections to improve the unit 


equipment readiness. However, when more than 11% of equipment modification 
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discrepancies are discovered, the SMAT should re-allocate its limited resources and 
analyze the poor inspection findings to discover possible causes such as administrative 
lapses or systemic issues. Figure 9 illustrates the value curve for the equipment 
modification status, in which the least value is where the metric score is zero and reaches 


a maximum value of 10 at a metric score of 11%. 


Equipment Modification Status 


10% 20 30% 40 50° 60% 70% 80% 90% 


Number of overdue modification / Total equipment required 


modification 





Figure 9. Value curve for equipment modification status 


2; Value Curve for Equipment Calibration Status 


The metric of equipment calibration status focuses on capturing the percentage of 
equipment calibration discrepancies during the SMAT inspections. This metric also 
reflects the operational status of the units’ equipment and ultimately influences their 
equipment readiness. Operationally, equipment calibration is mission critical to ensure 
quality, safety and performance of military equipment. Given that the equipment is 
required to be tested at different intervals, any significant findings may point towards 


potential administrative lapses which could pose safety and operational vulnerabilities. 


The value curve has a convex shape based on the relationship between the metric 
and the surrogate variable, where the percentage of PMCS discrepancies increases by a 


larger proportion with the metric scores. This is similar to the concept of increasing RTS 
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which is associated with marginal productivity as discussed in Chapter I]. Based on the 
empirical data, the metric shows little correlation relationship with the surrogate variable. 
Moreover, the data points of the units inspected are concentrated at above 57.7% of the 
metric score. This would mean that an increase in the number of discrepancies that the 
SMAT uncovers during an inspection would provide greater proportional of value in the 
anticipated improvement of the unit post-inspection. Therefore, the value curve can be seen 


to convex upwards at beyond 57.7% of the metric score as shown in Figure 10. 


As more equipment calibration discrepancies are discovered during the inspection, 
there would be more value derived from the inspections to identify the administrative lapse 
which would ultimately improve the unit equipment readiness. Based on Figure 10, the 
value curve has the least value at the metric score of zero and reaches a maximum value of 


10 at a metric score of 100%. 


Equipment Calibration Status 


10% 20% 30% 40% 50% 60% 70% 80% 


Number of overdue calibration / Total equipment required calibration 





Figure 10. Value curve for equipment calibration status 


3. Value Curve for Training Allowance Status 


The metric of training allowance status focuses on capturing the percentage of 
absolute surplus or deficit in the total training equipment assigned to the units. 
Operationally, training allowances are critical to the units as they signify the minimum 


amount of training assets required for training. 
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The value curve assumes a linear shape based on the relationship between the 
metric and the surrogate variable, where there is a proportional increase in the value derived 
as the percentage of training allowance discrepancies increase. Based on the empirical data, 
the metric shows moderate correlation relationship, with a correlation coefficient of 0.5. 
Moreover, the data points of the units inspected are concentrated at below 37.7% of the 
metric score. This would mean that the added value from findings beyond 37.7% 


diminishes rapidly. 


As more training allowance discrepancies are discovered during the inspection, 
there would be more value derived from the inspections to improve the unit equipment 
accountability. However, when more than 37.7% training allowance discrepancies are 
discovered, the SMAT should re-allocate its limited resources. Figure 11 illustrates the 
value curve for the training allowance stat, in which the least value is where the metric 


score is zero and reaches a maximum value of 10 at a metric score of 37.7%. 


Training Allowance Status 








Figure 11. Value curve for training allowance status 


4. Value Curve for Commodity Stage Status 


The metric of commodity stage status focuses on capturing the percentage of 
equipment left in commodity stage for more than 96 hours during the SMAT inspections. 


Operationally, the purpose of the commodity stage is to act as a temporary digital storage 
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account for equipment received by the units. The unit by regulation has 96 hours to move 
accountability for the equipment from the commodity stage to one of its other accounts. 
Units are advised not to hold equipment in the commodity stage for more than 96 hours as 
it would result in administrative lapse that would affect equipment accountability in the 


system. 


The value curve assumes a concave shape based on the relationship between the 
metric and the surrogate variable, where the percentage of PMCS discrepancies increases 
at a decreasing rate with the metric scores. This is similar to the concept of decreasing RTS 
as discussed in Chapter 2. Based on the empirical data, the metric shows little correlation 
relationship with the surrogate variable. Moreover, the data points of the units inspected 
are concentrated at below 6.7% and above 95.1% of the metric score with a low percentage 
of PMCS discrepancies. This would mean that the added value from findings beyond 
95.1% diminishes rapidly. Based on the law of diminishing return, there is a maximum 


range of value to which stakeholder can extract from the inspection. 


As more commodity stage discrepancies are discovered during the inspection, there 
would be more value derived from the inspections to improve the unit equipment 
accountability. However, when more than 95.1% of commodity stage discrepancies are 
discovered, the SMAT should re-allocate its limited resources and analyze the poor 
inspection findings to discover possible causes such as administrative lapses. Figure 12 
illustrates the value curve for the commodity stage status, in which the least value is where 


the metric score is zero and reaches a maximum value of 10 at a metric score of 95.1%. 
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Commodity Stage Status 


% 20.0% 30.0% 40.0 50.0% 60.0% 70.0 80.0% 90.0% 100.0% 


Number of equipment in commodity stages exceeding 96 hrs / Total 





training equipment allowance 


Figure 12. Value curve for commodity stage status 


5. Value Curve for Inventory Document Status 


The metric of inventory document status focuses on capturing the percentage of 
equipment with aged status and lonesome demands in the due and status file (DASF) during 
the SMAT inspections. Operationally, the DASF provides the order record of all equipment 
of each unit. All units are required to manage their records properly to have oversight on 
their equipment status. Document discrepancies may lead to accountability issues and are 


usually indicative of administrative lapses. 


The shape of the value curve assumes a concave shape based on the relationship 
between the metric and the surrogate variable, where the percentage of PMCS 
discrepancies increases at a decreasing rate with the metric scores. This is similar to the 
concept of decreasing RTS as discussed in Chapter II. Based on the empirical data, the 
metric shows little correlation relationship with the surrogate variable. Moreover, the data 
points of the units inspected concentrates at below 34.5% of the metric score. This would 
mean that the added value from findings beyond 34.5% diminishes rapidly. Based on the 
law of diminishing return, there is a maximum range of value to which stakeholder can 


extract from the inspection. 
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As more document discrepancies are discovered during the inspection, there would 
be more value derived from the inspections to improve the unit accountability. However, 
when more than 34.5% of inventory document discrepancies are discovered, the SMAT 
should re-allocate its limited resources and analyze the poor inspection findings to discover 
possible causes such as administrative lapses. Figure 13 illustrates the value curve for the 
inventory document status, in which the least value is where the metric score is zero and 


reaches a maximum value of 10 at a metric score of 34.5%. 


Inventory Document Status 


10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% 
Number of equipment with aged statuses and lonesome demands / Total 


equipment in the Due and Status File 





Figure 13. Value curve for inventory document status 


6. Value Curve for Parent and Child Relationship Status 


The metric of parent and child relationship status focuses on capturing the 
percentage of equipment with incorrect parent and child configuration. Operationally, it is 
important to serialize and manage equipment with parent-child configurations due to their 
intrinsic dependency which would influence their functionality. When the relationship is 
not established and managed properly, it has an impact on the timely conduct of PMCS 


actions. 


The value curve assumes a convex shape based on the relationship between the 
metric and the surrogate variable, where the percentage of PMCS discrepancies increases 


by a larger proportion with the metric scores. This is similar to the concept of increasing 
36 


RTS which is associated with marginal productivity as discussed in Chapter II. Based on 
the empirical data, the metric shows little correlation relationship with the surrogate 
variable. Moreover, the data points of the units inspected concentrates at above 50.5% of 
the metric score. This would mean that an increase in the number of discrepancies that the 
SMAT uncovers during an inspection would provide greater proportional of value in terms 
of the anticipated improvement of the unit post-inspection. Therefore, the value curve can 
be seen to convex upwards significantly at beyond 50.5% of the metric score as shown in 


Figure 14. 


As more parent and child relationship discrepancies are discovered during the 
inspection, there would be more value derived from the inspections to identify the 
administrative lapses which would ultimately improve the unit equipment accountability. 
Based on Figure 14, the value curve has the least value at the metric score of zero and 


reaches a maximum value of 10 at a metric score of 100%. 


Parent/Child Relationship 


10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% 


Number of equipment with incorrect parent and child pairing / Total 





training equipment allowance 


Figure 14. Value curve for parent and child relationship status 
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D. CONVERTING MEASURES INTO VALUE SCORES 


After the development of the value curves, the inspection team can convert the 
relevant metrics into raw value scores (RVS) for each unit. For operational security 
considerations, the MARFORRES units are given code designations in this thesis. An 


example of the inspection findings is shown in Table 3 for the unit code-named “R.” 


Table 3. | SMAT inspection findings for Unit R 








Equipment Readiness Equipment Accountability 








Unit (Code Name) 


Percentage of Equipment 
Stage Discrepancies 


Modification Discrepancies 
Percentage of Equipment 
Calibration discrepancies 
Percentage of Inventory 
Document Discrepancies 

Percentage of Training 
Allowance Discrepancies 





Percentage of Parent/Child 


Percentage of Commodity 
Relationship Discrepancies 





























& 


100.0% 


In this example, The SMAT inspection of Unit R has recorded an equipment 
modification discrepancies of 11.4%. Referring to the value model for this metric results 
in a corresponding RVS of 9.94. The RVS for the remaining metrics for Unit R can be 
derived in similar fashion as shown in Table 4. As described in the next section, swing 
weights are applied to these value scores to represent the importance of the metric to the 


stakeholder. 


Table 4. Raw value scores for Unit R 
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E. ELICITING VALUE WEIGHTS 


An additive value model is used to evaluate the total value that a unit gains from 
receiving a SMAT inspection. The value score for each metric is weighted and summed 
for each unit. Equation | is the expression of the additive value model for the application 


of swing weights (Parnell, Driscoll and Henderson 2010): 
v(x) = dizi Wi Vii), (1) 


where v(x) is the value that the unit x would gain from the inspection, based on the value 
scores for metrics i= 1 to n. wi is the measure weight of the i" metric, vi(xi) is the value of 
the score of the ith value metric for unit x, and x; is the score of the ith value metric for unit 
x. The weight assigned to the metrics reflect their importance to the stakeholders. The 
variation in the metrics’ range is based on data from past inspections. Figure 15 shows the 
matrix of the swing weights allocations in accordance with G-4 stakeholder assessments of 


the metrics. 
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Figure 15. Swing weight matrix. Adapted from Yee (2019). 
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For example, Figure 15 shows that the stakeholders have determined that 
equipment calibration status (placed on the top left) as very important for evaluating a 
unit’s maintenance health. Furthermore, this metric is also assessed to have a high variation 
in its measured range, which enables analysts to more easily delineate the relative goodness 
of a unit’s score with other units. Therefore, equipment calibration status receives a score 
of 10. The metric for equipment modification status (placed on the bottom right) has been 
identified as a less important measure and has low variation across its measured range, 


thereby receives a score of 3. 


Next, the swing weights of the metrics are normalized into weights on a scale of 0 


to 1 using Equation 2 (Parnell, Driscoll and Henderson 2010): 


fi 
= tt 2 
Wi pe ( ) 


where fi is the original swing weight assigned to ith metric, i = 1 to n metrics, and wi is the 
corresponding measure weight for the ith metric. Table 5 shows the computed measure 


weights for each metric. 


Table 5. | Computation of measure weights 





























Metric Swing Weights Measure Weights 
Equipment Modification Status 3 0.065 
Equipment Calibration Status 10 0.217 
Commodity Stage Status 8 0.174 
Inventory Document Status 9 0.196 
Parent and Child Relationship Status 7 0.152 
Training Allowance Status 9 0.196 

Total: 46 1 
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The final step is to assign a meaning to the final value score derived from the SMAT 
inspection program. This value score represents the SMAT program’s contribution to the 
MARFORRES mission. Yee (2019) adapted a performance matrix from the U.S. Army 
personnel readiness assessment shown in Table 6. The performance matrix has four levels, 
with Level 1 representing the greatest support to the MARFORRES mission while Level 4 
represents negligible impact towards supporting the MARFORRES mission. The thesis has 
adopted the same performance matrix to provide a common platform for evaluating the 


SMAT program. 


Table 6. Performance matrix of SMAT inspection program. Source: Yee 
(2019). 


Value to Unit Mission 
. Has significant impact towards supporting 
> 
8 to 56 veins Has considerable impact towards 
P supporting the MARFORRES mission. 


Has negligible impact towards supporting 
the MARFORRES mission. 


5 points and below 





) 
ab oh Has limited impact towards supporting the 
>5 
i 
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F. COMPUTING OVERALL VALUE SCORES 


Applying Equation 1, results in Table 7, the overall value scores of the 


MARFORRES units. The corresponding performance measure is derived from Table 6. 


Table 7. Overall value scores for MARFORRES units 


| RIES onwaraesine | vine ta 
po A STB 





The resulting value scores are checked with the designated surrogate variable 
(percent of PMCS discrepancies) to track the overall equipment readiness and 
accountability of the unit. It is logical to expect that the inspection program’s value score 
would be higher for units with higher PMCS discrepancies, meaning that the unit should 
be inspected. As a result of the inspection, the unit would assumedly correct the identified 
discrepancies and thus improve its standing within a reasonably short amount of time. For 
example, Unit R had a high PMCS discrepancies value of 0.81. Therefore, it is expected to 


have a significant number of adverse findings across the various metrics, resulting in a 
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relatively high value score. Based on the performance matrix in Table 4, Unit R has a value 
score of 6.5 which indicates that the inspection would have considerable impact towards 


supporting the MARFORRES mission. 


Although the value scores are observed to generally follow a logical pattern, there 
are some exceptions. For example, Unit U has a low PMCS discrepancies value of 0.07. 
However, it has a relatively high overall value score of 6.8, which puts it in the same 
category as Unit R. Given the SMAT program’s revised concept of operations focuses on 
virtual inspection, this has made the SR and MR from Yee’s (2019) study inapplicable. 
The need to find a new overarching variable for the unit’s maintenance and supply health 
has made it difficult to have a direct correlation. While adopting the PMCS discrepancies 
as a replacement for SR and MR, it is possible that there may be unforeseen confounding 
factors such as misinterpretation and verification of data that are initially difficult to 
discern. Since the change in SMAT CONOPS occurred midway through this study, there 
was no opportunity to other collect data to study the topic further. Overall, approximately 
70% of the PMCS Discrepancies and Value Scores pairs are compliant. Due to the limited 
duration and the difficult in collecting new data midway through this study, the findings 
are accepted. A major recommendation for this study will be to re-assess the value models 
and determine their consistency and robustness data that represents the actual performance 


of the unit in terms of equipment readiness and accountability. 


To verify the functionality of the value function, the post-inspection data of each 
units’ percentage of PMCS discrepancies were collected from 30 to 90 days after their 
virtual inspection. The units generally show improvement in their percentage of PMCS 
discrepancies values. For example, the post-inspection percentage of PMCS discrepancies 
value of Unit R improves from 0.81 to 0.76 within 30 days after inspection. At the time of 
writing this report, post-inspection data for other units was not available. However, the 


improvement indicates the utility that the value scores can provide. 
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G. OPTIMIZATION OF SMAT INSPECTION 


Chapter III discussed the procedure to develop the SMAT inspection schedule for 
a six-month period using a greedy heuristic to optimize the inspection schedule. However, 
the objective function was based on maximizing the total value score of inspection 
schedule. In general, the greedy heuristic is how the SMAT is implementing value scores. 
Considering these 23 units, the greedy heuristic scheduled eight units for inspection in the 
six-month period and attained a total value score of 47.3 as shown in Table 8. However, 
approximately 40 out of 100 available FTE were unused. These results highlight the 
principle of the greedy heuristic model; the model selects a locally optimal solution (unit 
with the highest value score) for the given month of the inspection schedule. The model 
will choose the unit with the highest value score that can be inspected with the available 


FTE in that given month only. 


Table 8. Inspection schedule using greedy heuristic model 





An optimization model based on Lugo’s optimization model and focusing on 
maximizing total value scores resulted in scheduling 16 units for inspection in the six- 
month period and attained a total value score of 84.9 (Table 9). In this schedule, all SMAT 
available resource were used to gain the most value from the inspections. Compared to the 
greedy heuristic model, Lugo’s optimization model is able to attain approximately double 


the total value score. 
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Table 9. Inspection schedule using an optimization model based on 
Lugo’s (2020) optimization model. 


| U | Month | 68 
| O | Month2 | 69 
| Bo | Month3 | 3.4 
| S| Month3 | 7.4 | 
| oT | Months | 5.7 
| Q | Month4 | 64 | 
| P| Months | 6.2 
| Ro | MonthS | 65 
| D | Monthé | 5 
| S| Monthé | 6.9 

| 84.9 


Total Value Score: 























The results of implementing the value scores in conjunction with an optimization 
program, provides evidence for how the G-4 can more efficiently use its resources. 
Through this process the G-4 SMAT can obtain the set of inspections that will provide the 
greatest contribution to the MARFORRES mission. 
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V. CONCLUSION AND RECOMMENDATIONS 


A. CONCLUSION 


This thesis examined the SMAT inspection program that the MARFORRES G-4 has 
developed to support the organization’s mission. First, this thesis reviewed the value models 
from a previous study of SMAT operations. The existing value models contained some 
inconsistencies and ambiguities for interpretations that would fail to reflect the value of the 
SMAT inspection program. Next, this thesis sought to modify and validate the value 
models to more accurately capture the contributions of the SMAT inspection program to 
the MARFORRES mission. Lastly, this thesis described how the application of value 
scores in an optimization model may assist in resourcing decisions in a budget-austere 


environment. 


Based on the SDP framework, this thesis reviewed the existing SMAT objectives 
hierarchy to verify the stakeholder’s needs, and to assess the value models. The objectives 
hierarchy leads to the attributes and the corresponding metrics, which must 
comprehensively account for all key objectives that are important to the stakeholders. 
Through the stakeholder analysis, the top-level fundamental objective was verified as being 
focused on ensuring that the MARFORRES units have the right types and numbers of 
properly maintained equipment. The second level of the hierarchy continues to focus on 
MARFORRES G-4’s interest. The sub-objectives were revised to the following: 1) 
Improve equipment readiness; and 2) Improve equipment accountability. The sub- 
objective “Improve impact on unit processes” was removed since the SMAT has changed 
its concept of operations to focus on virtual inspections and no longer requires the units to 


submit corrective action plans. 


The sub-objectives are subsequently used to identify the key attributes and 
associated metrics that would allow the SMAT to evaluate the equipment readiness and 
accountability of the MARFORRES units. The quality, utility, or relevance of some 


variables made them unsuitable for creating the value models. As a result, perpetual 
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inventory status, MPR status, and PMCS status were omitted from the existing objectives 


hierarchy. 


Given the SMAT program’s revised concept of operations focuses on virtual 
inspection, this has made the SR and MR from Yee’s (2019) study inapplicable. The need 
to find a new overarching variable for the unit’s maintenance and supply health made it 
difficult to have a direct correlation. The existing variables of SR and MR ratings are 
replaced by the PMCS discrepancies as the surrogate variable. This thesis finds that of the 
23 observed units approximately 70% of the PMCS discrepancies and value scores pairs 
are compatible. Due to the limited duration and the difficult in collecting new data midway 
through this study, the findings are accepted. The PMCS discrepancies provides an 
indication of the overall maintenance health of the organization and guides the 


development of the value curves. 


Next, the thesis attempted to represent the value derived from the performance of 
each attribute with a value curve through quantitative value modeling. The surrogate 
variable, the percentage of PMCS discrepancies. is plotted against each metrics to develop 
the value curves. There are three main shapes that are hypothesized in this thesis: (1) Linear 
RTS, (2) Decreasing RTS (concave curves), and (3) Increasing RTS (convex curves). The 
thesis verified the relevance of the value curves through theoretical, empirical, and 
operational justifications. Furthermore, the post-inspection data of each inspected units’ 
PMCS discrepancies are also collected to verify that value models meet the objectives. The 
inspection findings of the metrics of each unit are then converted into RVS using the value 
curves developed. The RVS are subsequently transformed into WVS using the measure 
weights to represent the importance of the metric to the stakeholder. This WVS can then 
be computed for the total value score of each unit, which represents the value that the unit 
would gain from the inspection and thereby the value that MARFORRES would gain from 
the unit being inspected. The value score will serve as basis for SMAT to make better- 


informed decisions on whether resources should be invested to inspect the unit physically. 


The thesis also demonstrated the application of the value model in scheduling 


SMAT inspection for a six-month period. A greedy heuristic model and Lugo’s 
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optimization model are used to develop the inspection schedule to maximize the total value 
that can be derived from the inspections. The principle of the greedy heuristic model 
focuses on selecting locally optimal solutions (units with the highest value score) in each 
month. The greedy heuristic was able to schedule eight units for inspection in the six-month 
period and attained a total value score of 47.3. In this schedule, approximately 40 out of 
100 units of the SMAT resources were unused. Lugo’s optimization model was able to 
schedule 16 units for inspection in the six-month period and attained a total value score of 
84.9. In this schedule, all SMAT available resource were used to gain the most value from 
the inspections. Compared to the greedy heuristic model, Lugo’s optimization model is 
able to attain approximately double the total value score. The results of implementing the 
value scores in conjunction with an optimization program, provided evidence for how the 
G-4 can more efficiently use its resources. Through this process the G-4 SMAT can obtain 
the set of inspections that will provide the greatest contribution to the MARFORRES 


mission. 


B. LIMITATIONS AND RECOMMENDATIONS FOR FUTURE WORK 


The SMAT program’s revised concept of operations means that there may be 
certain unforeseen confounding factors such as misinterpretation and verification of data 
that are initially difficult to discern. The exact explanation for this abnormally is not 
available due to the limited duration of the study. A major recommendation is to allow the 
value models to be re-assessed and improved on for their consistency and robustness 


through a more representative measure for a unit’s equipment readiness and accountability. 


The sub-objective “Improve impact on unit processes” is removed in the revised 
objectives hierarchy based on the new SMAT guideline and due to the outbreak of COVID- 
19 that limits SMAT’s ability to physically assess the degree to which the inspections 
improved the units’ processes. This sub-objective can be re-assessed in future as it 
encompasses another aspect of inspection that focuses on the developmental impact. 
Various metrics such as the percentage of corrective actions implemented can be further 


evaluated to examine the extent to which the efficiency of the unit processes has improved. 
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APPENDIX 


Specifics of the Value Scores for MARFORRES units 
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